
Need to be testing the tool. And and throwing our questions about this information. And until Um, we have a few more questions and some of the things you say will end up fitting under some of these headings. But don't worry about the headings. It's going to like try and pick up what you say and pop it in.

And also as you heard Emma's going to be taking some notes as well. So we can try and captures much rich information as we can, and then we have the recording things as well as so we can always go over the deceive, we've missed something because sometimes you can never tell what you meant when you write something compared to what it sounds like.

When someone says it. Yeah, the information that gives the way a lot more than the text. So, Went to ask. You was really a broadly. It was this effects of misinformation. When when you think about, you know, the effects of misinformation, What do you think? Could come from misinformation?

What can it lead to? Fact decisions. Bad decisions. Yeah, General, Ms. Trust, yes. It could be it can start from this is a small general mistress area of this but then how does it grow, you know, how does it balloon up? You can. Then create more stronger opinions than you can go down other paths and then how it can lead further on from there.

So mistrust and like going down past, yes, that leads to what greater? Yeah. Beauty greater. On one say misfortune about everything, that's the right word, you know, maybe more even further, bad decisions, you know, and stronger opinions and a greater divide. Extreme. Yeah. So you heading down one end of a spectrum and I don't you extreme cases.

It can be very dangerous as well. Yes, just thinking of an example like if Allowing Tanzania from allow young. We had misinformation on our business people in our business. Yes, yes, that misinformation and unfortunately let the deaths. Yes. So the outpeople who had Albanism became sort of targeted. It didn't like it, So, yeah, and nothing will always I think we'll always not alone either.

I think it happens another places too. Yeah, yeah, that's really difficult. My country. I can remember. We had somebody in and he got unwell with everybody and it was fine but I can imagine easily not be okay. Anything else that you can think of any specific? Any specific occasions, you think women's information has led to something God awful.

If you got examples that you think of when you think about that, He get Bryant sometimes like right writing. Yeah. Due to like misinformation. Yeah, yeah. It's like, can really boiled right over to civil unrest? I think anything really that could consider contentious from About anything within the past 10 years, whether that would be Brexit covid.

American presidential campaigns on regard to other countries as well and how that misinformation can be. Constricted how we can see that it has to be one way or the other way or not. And there's no in between you know, that that line doesn't get bloody head. You happen to be that way, all that way and there's no other choice in between you know and bits from this information.

It's focused on this path or focus on that path rather than taking the subtleties. It sounds like, am I right? You discovered me like polarisation. Like people becoming more. Yeah. There isn't it? Browns and polarisation from that misinformation, you know, and how that misinformation can then. Exacerbate, that certain point of view before that polarisation whichever way.

Respect you, right? Up here else. I'd say that's like, extremeless kind of like times into that kind of view. Yeah. Which is usually quite dangerous. Yeah. Yeah. You're not trying. All right, so Um, when you're looking at a piece of information, reading an article, something in the paper or online, how, how do you check the accuracy of information?

What do you do if you do anything? What do you do when you're trying to? You know because when I look on line like information is usually on social media like Facebook or something. So if I wanted to like check it this actually like true I'd go like official sort of like website rather than just like social media because it's a lot of misinformation and like Facebook and nighttimeness place.

So you'll check another source. So you could you look specifically for kind of saucy looking for maybe like government source, it depending on what kind of misinformation it is or possible in this information, okay? Yeah, that's good. And then just don't trust it at face when generally. So anyway anything you read anywhere don't trust the face most things see?

Okay you cross check with other sources. And then you kind of how to decide for yourself what you're going to lift from that. Yeah, see what is I mean I wonder and tell me what you think when you're doing that kind of cross checking, how do you decide? Okay, if it's there, that means it's real or could it be that?

They're also fallen for a fake story and now it's face story being spread on the sides that I would normally think are. Okay. Do you ever find yourself in that position and you've checking that you wonder whether or not the thing that you're using is a validating site? Yeah, it's I've noticed.

I think this is gonna happen. Quite a lot, is with a large immediate groups, they have smaller media groups in between them and are they cross reference in for me, each other. So could be outside three or four sources here and they're all the points the same thing, but maybe it's all coming from the one, the main source that's been trickled down and Second accuracy, it can be tiring.

You know? Say oh I've got check from here but there's a resource from here. I'm gonna does that then link to listen, does that link to that and then generally Are people that bothered, You know, they just want to go, this is my news, you know, and then want to get on with the rest of their day.

They don't want to be going down 15, 20 minutes but it's not just just want to skim it. You just want to get information and get the information. It needs going out and don't want to be, you know, I don't think people are interested in going through and researching their resources and going through that.

I just don't see that. Yeah, interesting. Majority time to, or don't want to put. I think it's a bit about, don't want to put the time in all, you know, what's their level of priorities. And I think fact checking some new sourcing information whether It's already coming from use that maybe tweaks, that point the view I don't think there'd be better.

Okay. No, that's good. That's a good point actually. How much Effort. Yeah. Two people want to put into fact checking for themselves and looking for accuracy. Only fact, check. When it's something important that or something. I actually need to know whether it's true or not, Okay? It's just like general gossip than I would just ignore it.

Let it slide. Yeah. Yeah. That might not be and then move on. Yeah. Um, so what kind of information do you think is most prone to misinformation? So like is it sources or there certain A media sources that you think are prone to misinformation or types of media that are likely to be.

Some targeted with misinformation. I would say social media and general because a lot of people would just voice their opinion kind of like 10. They know what they're sort saying, but it's just some spreading this information. Yeah, yeah. Um, any particular like either social media sources or other sources specific that you He kind of looked to, and you think, oh, I tend to not be sure with that one.

You know, there's a lot of this information on Facebook, when I look at I'd say there's a lot misinformation on YouTube so Facebook and YouTube. I think I sing from Nexus, so easy and just to post that video or post out the stream and go work, okay? So, Who's behind this and you know, go back like If they present this as a certain point of view in the made this new source which is always going to have the certifiest point of view and you know what the reasons and you know maybe maliciousness behind it, you know and Know if the queen from something new, this is what they say.

They can be in, they can be an expert. But, you know, the, you know, because of maybe just got connections or something. Of like that you know how a level is that aggressive going to be, you know what's the what's the truth behind that truth? And you know, maybe they're just making this new source of and then of course referencing for themselves for that certain point of view and it's like in it so easy to get that information out there and it can be with the algorithm who's got the most money from, who's got the most financial backing us referencing themselves.

Yeah. Yeah I like that. That's good point. So people can generate a lot of hype from like some fake accounts or yeah and I say anything from as well as who's got the most financial backing behind because you know they can pay to be the pay post social media post they can pay to be the top of the algorithms, they can pay to put their opinion forward in the more people that see at the more people that then believe that and then it conspire them just perpetuating itself, isn't it?

Hello. Okay. So using the tool, When you were using our Paper tool. What did you did you find it? Easy. Did you find it? I realise that you can actually click the things because I didn't realise. You can actually click again, that shows you the stuff. I was like, I just looking, I just saw number.

So I thought that, like, oh, you could click it and then the highlight and more cameo. Yeah, but it took me about, okay, so that's not obvious. Yeah, I didn't think it was obvious side this, so that's useful that you didn't need. The once you did click on it, did you find that helpful the additional information like helpful than just seeing, just whatever.

Yeah, Saying this is something about actually worked on quite recently done with all the tools with UX. So I might go into some really bothering points here. To me, this should be more focused on colour, gradients and scaling. It was some concerns, I had around. And I think it's maybe this is a stuffer down the one though because that is what we were interested in things like accessibility for people with visual impairments or something like that.

This could have been nice of going back from the arrows and some some nice little animations tool tip dropdowns modals. A decimal points, always didn't make sense, because I think it could have been a more visual as a percentage, but then you've got some cross translations, you can do with that.

And sometimes as well to me the my main issue with the tool I had in general, is it wasn't enough in my face, you know? So I'm going through reading articles and articles and artists, I could easily forget that that's running. Yeah, yeah, you may not know just pop in.

I wouldn't notice the results and the misinformation would still Feed in. But then, I've not I'm not consciously checking it because I know it's in the background and then it's saying goes back to the effort of And click pop up, go down, go review. It maybe even if things have you know, certain sections or certain sentences, they could be grouped together so you could go.

So it's got this score overall for this this and this and So this this clicking to make it show, what it's done is getting in the way. Yeah, I think it's more for when I'm reading shop, I'm going through reading news. I want to focus on the news and The tools are not too.

Two out of the way and then when you want to check the information, it's too many. Clicks. It could be more aggregate, certain scores. If it's certain sentences or, you know, call escape colour scaling will gradients or something. More about gate more in your face. It's that makes sense.

Another question, I guess alongside that would be, would you want to see something in a margin? It would you want to, you know, to me like Adam and it shows that it's there. I think it's more. There has to be a clear visual tool on the screen wherever that's in a marginal and that's highlighting sentences or highlighted to certain bits of misinformation.

I don't think Twitter's done something recently, they've ever miss information on tweet. There's a pop-up box underneath now that says, this is clearly wrong, I'm going, they can cross reference of like that has to be really. Face. Easy. He? She's jumping that down. No, that's really helpful. Actually, and hear a list and the things that you're going to be today.

Well, yeah. And the list that you stream down really quickly, I'm glad we call it on on the recordings. It is interesting that you've mentioned about the accessibility for people with visual impairments because something we've been thinking about a lot, you know, within that colour coding and things like that.

But how could you because everyone's familiar with like a traffic licence too, so something like that, but We? And we were wondering about, you know, is there a particular symbol or something? That might do it, the arrow thing might be, I think same, it's more which design philosophy. What's going to go down with materials?

Should look like human computer interaction, but you're also to me, have to be careful with the traffic light system because of certain visual impairments. You know, black textures are going to read while I'm red. And then, you know, or the basic gonna be Understandable, from different nationalities or perspectives or things like that is, you know, it's Yeah, it's in trustwit system might work in general for let's say British people here but for someone in a different Country complete.

Different mindset philosophy. Well like what the colours actually mean as well because red could be anything for someone. Yeah could you stop or danger or and yeah absolutely money and look. Um yeah, probably the wrong message. You. So we would, sorry, I just gonna say, we just at this point, we're talking about what you found easier, difficult?

And some of the UX stuff was tricky. I mean, did you find any problems? Didn't know what the Numbers of percentages meant initially. Yeah, I still don't. Still, that's fine. Yeah, they're red. I just assumed was something. A bad. Okay. Yeah. So for example, Yeah. It's objectivity or something, or something that feels red.

I think that that article was Not very subjective or something like that. So yeah, just the colours and the numbers. But also, I didn't know what. I should be looking for under each heading. So, first person, I think the first question I asked, was a ways first person? Yes, bad.

Yes. Why is that considered? Yeah. Up. Yeah, so not just the definitions but the reasons you need to know the reasons why those headings have been chosen. So what is it about? Those headings that make them markers for missing information? Yeah. Yeah, that's a good photo education on that.

So if you had an explanation for each thing with a name is explosional, what it is? And why we might be interested in seeing it would you want that integrated into the tools so you have sort of? I don't know a question marks. You can expand it from there or do you think actually that we've been better just to have a main page that you can navigate with all the information?

Welcome could be helpful. This should not have to leave the page. Yeah I think some kind of tool tip mode off with a link with them provides more information for that would be a really simple thing to implement my really beneficial Okay, anything else? Stand out is being tricky?

Difficult of talked about this, the colours, the scales. The having to click when maybe it'd be easier if it was already. Visible having it more in your face. So that you don't miss it Is anything else that we haven't mentioned that you think would make the tool better because it is tricky.

They need to see because the definitions Definitions. Yes, of course. Yeah. And think for me in them links in successibility, but the supply screen readers, and no, making sure it's got certain elements and things like little work with screen readers. Yeah, yeah. Absolutely, that's true because we think does does it?

Um, Know, it's a more information that you would have wanted with talked about the definitions. And the reasons why those headings are there Is anything else that would have helped you understand what the tools doing more or just get more. Before utility out of it, you know, get better use that of it.

So you would have wanted to see that we didn't present I think for me, one thing I think will be quite interesting is use of aggressive language for bias. Um, yeah, same day it's a certain keywords. It can pick up to have a certain it's really going down a certain point of view for that certain level of aggression to make a certain point to make a certain level of mistrust or something like you, their muslim certain words and no, how that aggregates together, you know to have that take over.

How bias is this article whether it's what way or near just do that. Use of them words So aggression. I mean, I don't know where you've looked into. Wearing. Yes. Swearing is a swearing is one of them. A big thing, I guess a lot of that come under the subjectivity.

Thing. I think sometimes put situation as well. Sometimes kind of like the examination marks that kind of yet you're not wrong at all. That definitely came up as yeah, as but we don't have capturing that. Yeah, we is like extra {exclamation-point}. I didn't see it was something was difficult.

Well that's what. We should put it on the list for next time and the list and make you mean we Thanks for Jeremy, We love Santa are in lovingness, But that's that's a good point. Because if we can capture exaggerations in that respect you know, whether it be through excessive aggression.

Oh, excessive excitement. Those all markers. So it's about getting a tools to pick, you know, knowing what it's doing to pick them up, half the problem that we we encounter. Helping it to find the things you want. And anything else that you would want to see the tool do that, it doesn't currently do.

You know. Any concerns when you're using it. Did anything worry you about the tool, whether it be for your use today or your thoughts about future, And that's the some of the words. It highlighted, wasn't quite accurate, like yes, yeah. Okay, yeah, what did you spot? You It was like I think one of the comparative it was like saying longer highlighter, but it was in the sentence, it wasn't used as like a comparative, It was just like saying it was no longer thing.

So it wasn't yes. Yeah yeah. It wasn't a magic too. Yeah It was a couple others as well which I knows is because I studied English language at college. I kind of so you pick up on those things. Yeah. So I know like what's on. That means anyway. Okay so that that ER again I think no work very well.

Did you find down transgender? Yeah, one. That's because it picked up as well and we think it's the ER It's the yeah it combined with something that adjective and it is with any R&B. So it's like oh I think that's That's not. Yeah, that definitely needs to be fixed.

And anything else. Any other concerns? It's got some errors in how it's picking things up, which we need to Hammer down a bit, anything else you can think of the concerns, you Don't think it was concern me directly from anywhere else in store that extension is, what with doing with that data.

Are you? Hybridgating for everybody who is there, a certain, you know, I'm sitting down to, you know, if you look in a certain articles and you might want to go certain ways does the tool then configure that and then maybe contribute to go get these just to make sure that tools from a neutral.

Neutral stamp. And whether that's just a set of link and just provide some extra information, just just make it there. Okay. Yeah. So the you want a bit of background on. What's the tool doing? How's it working? What to do with your data? Yeah I think so if it's it's long as it's there to be viewed even if people don't want to use it, now it's there and I think that would be beneficial Hmm.

I mean I should say to you background a little bit of background is that one of the features of the tool and one of the reasons that it requires you to, to a bit of clicking is that it's giving you the opportunity to look at what's been analysed. Look at the results that it has come up with, and it's only the results that would be fed to research if you want to.

So you get to decide. So, even if you join this study, that was using the tool. And you are one of the participants and you were doing all the things that they said to do and you got something. We got actually I don't want to share that you don't have to so that would that is one of the features of it but we haven't made that explicit today.

I think that's interesting makeup. Okay, more explicit. Yeah. Yeah, it's more of us doesn't work, But it does open those questions up of like, but Like, you know what you doing with this thing? And it was the the reason it was devised was really about giving the opportunity for a researchers to only see what they needed to see.

I think that was one of the primary things so that we don't have to look through. Everybody's information, just capture the bits that we are after Um, and some, that was to avoid researchers from seeing things that were distressing in some texts and upsetting Um, and also giving obviously the use of the opportunity to make those choices and not feel as if once you're in the studies.

No choice. But People are sort of thought. Well, if if people are posted this online, then it must be okay for me to use it. But if if you post something on Twitter, you intended it for Twitter. You didn't intend someone to go take that and do something else with it and there's this bit tricky that don't.

Yes. So that was the reason we. Yeah, like what people expect to be done with their data legal but is it okay, is it okay to just troll every once it's not really So we try and The way you need to do. Yeah, do you trust the tool to flag language?

Often found in misinformation? Up some more somewhat obviously. Yeah, so long And trusted to the interested to flag language often found in Miss in misinformation. So is it doing what it thinks it's doing? Do you think it's doing what it says to doing? I'm not too sure. It does point out good things, but I think you mentioned the Exclamation marks punctuation?

Yeah more of that because yeah. So you wanted to be even more kind of like targeting mode, more more markers. Yeah. Because if it's missing stuff out, does that feel as if there's a chance, things will slip through? Yeah. Okay. Because those tend to be the articles that I know, but the five exclamation This is my email.

Options. Love it. Lovely? Yeah. Yeah. That's really good. Okay, so given the kind of conditional response, Yours is so you said you want more things to be targeted. So you feel confident, What do you think is lacking? That makes you go somewhere? Well it's just accuracy because obviously is playing out stuff but isn't true.

Okay. So it's those kind of like the the ER thing is it's getting that, right? Yeah. And then you need more accuracy before to be more trustworth. So more, you accuracy and more targets And I think for me as well, is there's the repair personal bike. So I think, when I look through Robin and checking tool out, so it's evidence here and so that first and is there a bias ground?

My end up disagree with the two, like you've got consider as well? The same time, you know, the bits like, oh yeah, picked about every fight. I did pick up on that. On all I haven't seen tall on that and just know, does it make a mistress in the tool to get?

You saw something that it? Yeah. And then it's just more making sure that it's just under tools in the United States and then that could be wrong down the line and I guess that's part of the information to give people is that it's still development and maybe there's a feedback loop that needs the built into the tools to say, if you identify anything that you feel, maybe,

A little bits that are not quite right. I think people would have more trustive than you that we were continuously trying to improve it, and they could involve themselves in that. Than these two. Yeah instead of it being a wall you know just look at what's on the wall and that's it.

You got no say to try and make it better. Because all the things you're saying, would be the kind of things. If we had a tool running, we'd want people to be able to let us know, some you can improve it. Um any factors that would help you trust it more.

Is there anything else that you think would help you? Besides what all you know you can you can leave at the points if there are any there that you think really important or is anything else that would help you. Have more trust in it.

I think for me, I know the house, I think it's just the stage with the tool, but I'd like to see 15 favourite extension of it because That's in their nose, kind of work, it's interesting texting articles but it's a case of, you know, something like say Of the misinformation, we could talk about social.

Media fees. How does it pick up on that in the north certification? Short form takes to the long form, text still, you know how it integrates into other new source and out of our outlets from there and you know, make taking it off from just a extension, how it is?

It further develop across all all your platforms and where you get me the news. Whether it's the web browser or social media old mobile phone apps, or anything else, does a couple of things? They're so there's something about what form you look. Yeah. And then there's something about how the sites work.

Yeah. And I think also as well, it's In these articles and how that can be used as part of the misinformation, like, is this image? That is the article, not like, say 90% correct. But then the image, she's exactly exacerbate. The point is completely from some of the writing media aspects as well, which isn't currently not there.

So we're not capturing images voices or anything like that, but yes, they all, they all have a part to play the the hot, the, the pigman image. If you ever want to check it out, is pretty impressive. Have done to make Pigman. Yeah, it's absolutely. If it makes it more convincing for me but for me.

No, it's just funny. Yeah, I mean, I don't believe it anymore by seeing the picture. I just laughed. Maybe someone suggest yesterday. It wasn't the picture. That was that the picture itself? The first picture was added credibility to it. So many features of this pigman. Yeah, deadly starting not to believe it because there were too many images of something.

Have every single pigment isn't he's not realised. But somebody's been a cracking jump, making one. Yeah. Little baby essentially, did you? We didn't say that, by the way, that all of those. Oh, I think it came out that early on, I think Jeremy. Yeah, yeah, none of them are really accurate.

It's always. Yeah, it's all look up. No big man. Sorry. Um, So, I've already asked you about what information. Well, what factors would help you trust? Is there more information you'd want? We've had definitions. What else would you want Every related articles Related artists across chicken cross? Check, I love.

Lovely. Yes. Okay. So going back steps and seeing where did this information? Maybe come from or maybe just what other articles are around that are of the same subjects? Yes. To see what they're saying about it. Yeah. Because we could line up some suspects couldn't. He said, how does this compare?

It's all of those different articles. Yeah, yeah, like that. That's really good. It's like the detective work. Anything else any other information you would want to see that would help you in your battle against misinformation or help the tool to make that easier? Up, maybe like a dates as well, because lovely, lots of information gets updated or changed.

So it's good to know like when it was like published or when it was updated. So you know what, what the latest information is. Okay, yeah. So versions versions of it the dates, when it came out, where is it? Come from and what other articles are like it. Nice.

What else might help you if anything? State.

Is anything you'd want around the measures. You can I know you've alluded to it already. Some things about measures, anything else around how the Results are displayed. So obviously, at the moment you've got the little headings which we want more definition for, they have a score next to them.

Each of them has a school Um, is there some other way that we could present that might be easier for you to digest or have more information than what's currently there? Last? You know, he say, what the scores out of cuz you don't know, like what the number like, how?

Like, how fast am I going to? Millions was like oh yeah, that's nothing. It like a metre. Uh-huh. Say or You know, the I've got the effect of like something objectivity. Yes. Objectivity, I guess. So, on somewhere on a lines that you have that interval? Yes, yes. You, that's good?

Yeah. So you can see where it's sits on the line. Anything else that you would like to see around the way that the results are presented to you?

You can say what? Like, what school you're looking for like because obviously, if it's like a high school, what does that mean? Does that mean it's more like like this information that's? Yeah. Yeah. It's a high school. Good. As a high school, It doesn't actually tell you. So yeah, yeah, okay, so yeah.

Sunshine of that. Some So what what does the school actually mean as far as good or bad goes? Yeah. Good point because some of them isn't necessarily. Yeah. No, no, I don't know anything. The language doesn't. That's like, I know the word. I'm not sure about schools. Here we falling his eyes.

I thought. I mean, the last thing's really questions around. What more information would you want? That's not provided by the tool, or do you want it? Provide the different ways. It's kind of reiterating the questions of already asked you. But if this anything else that you can think of you that you've not mentioned yet.

The nickels you or you think or if I if it was if it was me I would have made it like this and please. And that's the speed is ready, quick. I don't know, it's because the articles were really short. So I was wondering if it was a really long ark or how long would it actually switch the done?

Yeah. Because then you might not bother away that long. Okay. Yeah. So will it always be like instantaneous? No. He won't get much longer. But you'll be a big longer, So I could be another second or two. I think we can expect of this time last year and it was like a minute.

So believe it or not, it's improve. Next episode, It's got many, many steps to go, though. Anything else? I think one thing for me is how well the tool pick up if it's using some kind of Visual. Code inside the language that would then trick. The algorithm. So I think I said Set like saying setting characters that could be used and set languages.

Then does it miss for misconscrew that within the article? Or let's say, they've gone from certain that unique code inside the language, that then makes it appear human human readable, but then just that trick, the algorithm behind it, which then, you know, can then create that bias. Because, no, for the top 10 towards the tool like that, how do they buy past the tools?

You know, it's obviously you always been fighting each other, it's things like that. That's good point actually. Yes, so people are going to try and hack Of these types of things And it is going to be by playing with languages. Yeah. Yeah. To be able to fool it. Know into kind of maybe skipping the whole bunch of tests.

So I don't think I don't sense works this, we might read it's just a little bit load of unicode or something different like that. But then you know when it's officially put together it to the human just looks like a normal normal work long text. Yeah. Bypass the algorithm and posting things that are against the community standards but they're using.

I see lots of misspellings to do things like that. Yeah. To buypass with the triggered that trigger the algorithm. Yeah, it's very it's creative. Means the humans will always by pasta system, doesn't work for him and aligned Unliked. Yeah.

Corn. And So it's anything else. But you want to add the tools only use for like English articles isn't it At the moment? Yeah, it's only set for English. It was another other people have mentioned like about other languages which just about just about getting it to work in English.

One day. Yes, yes. We're gonna need a team of people to help us and just like change it. But then you know Google Translate might do a lot of that. Happy lifting for us for all I know. Actually thanks. That's no, That's a no. So we will need a whole team of linguists.

More than 0. It definitely. It's interesting, anything else? When shout now. And if there isn't then I'm gonna thank you for your time. Video. Thank you for you Today. Thank you so much, And so useful. Yeah, yeah. That's really great. And I can tell you some treasure back there.